Let’s Play<tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge lexical polysemy remains unclear. We propose a novel experimental setup for analyzing this in LMs specifically trained different languages (English, French, Spanish, and Greek) multilingual BERT. perform our analysis on datasets carefully designed to reflect sense distributions, control parameters that are highly correlated with such as frequency grammatical category. demonstrate BERT-derived representations words’ level partitionability into senses. Polysemy-related is more clearly present English BERT embeddings, other also manage establish relevant distinctions between words at levels. Our results contribute better understanding of the encoded contextualized open up new avenues semantics research.